Capstone IT is seeking an experienced Lead Data Engineer to join one of clients' Data Platform team to build scalable enterprise data solutions that drive strategic business outcomes.
As the Lead Data Engineer, you will partner with Product Owners on strategic initiatives, understand business and functional requirements on projects, provide hands-on technical leadership to the engineering team with project execution and ensure deliverables are delivered with high quality and on time. You will help lead the iterative design and development of our cloud data warehouse. You will be responsible for designing, developing and maintaining data pipelines and data in our data warehouse to support Data Science, Business Intelligence and business operations. You will lead and guide the data engineering team, ensuring adherence to quality, standards, processes, and methodologies. Additionally, you will support the team in overcoming any technical challenges they encounter.
Duties & Responsibilities
- Work closely with Product Manager and business stakeholders and develop a strong business understanding of our clients strategic and project level data needs.
- Work with Product Owners to understand and translate business requirements into technical specifications and logical & physical data model requirements.
- Design and extend data models working closely with Product and Data Architecture.
- Design data pipelines that process both real time and batch data leveraging modern data engineering best practices.
- Build, implement and test data pipelines using Databricks, Azure Data Factory, SSIS and other data integration tools.
- Take technical ownership of projects and ensure deliverables meet scope, quality and timelines.
- Provide technical leadership and guidance to the data engineering team, including mentoring, coaching, and fostering a collaborative work environment between all team members.
- Support team members with troubleshooting and resolving complex technical issues and challenges.
- Implement coding standards, conduct code reviews to ensure proper code quality level.
- Implement effective source code management and CI/CD best practices to ensure data pipelines are automatically tested, integrated and deployed to production.
- Partner with Data Architects to implement effective data integration and storage design patterns. Contribute to continuously improve existing data architecture and frameworks.
- Assist in the development and implementation of data governance and management solutions.
Required Experience:
- Bachelor’s Degree or higher in Computer Science, Engineering, or related field required.
- 10+ years of experience in data engineering and data warehousing.
- 5+ years of cloud data engineering and big data experience .
Required Knowledge, Skills And Abilities:
- Experience with cloud data engineering and data warehouse technologies in Microsoft Azure, Google or AWS. Experience in Azure is highly preferred.
- Strong programming skills in python and SQL for data processing and automation.
- Strong technical skills in big data and Data integration technologies including Spark, Kafka, Databricks, Azure Data Factory, SSIS and SQL.
- Experience with designing and building both cloud and on-prem data warehouses. Experience with Lake house architecture is highly preferred.
- Experience in data modeling for transactional and analytical needs. Experience with NoSQL databases is preferred.
- Development experience in Microsoft SQL Server including experience with tuning queries to optimize performance.
- Skilled in Visual Studio with the use of source control applications such as Azure DevOps.
- Experience with CI/CD and Jenkins.
- In-depth knowledge of data management principles and best practices, including data governance, data security, data quality, and data integration.
- Experience with delivering data solutions iteratively using the Agile Scrum framework and development processes.
- Experience in providing guidance, mentorship, and technical support to a team of data engineers.
- Demonstrated ability to troubleshoot and resolve complex technical issues related to data processing in a cloud big data infrastructure.
- Exceptional analytical and conceptual thinking skills coupled with strong written and verbal communication skills.
